Emotion resonance and divergence: a semiotic analysis of music and sound in 'The Lost Thing', an animated short film and 'Elizabeth' a film trailer

Music and sound contributions of interpersonal meaning to film narratives may be different from or similar to meanings made by language and image, and dynamic interactions between several modalities may generate new story messages. Such interpretive potentials of music and voice sound in motion pictures are rarely considered in social semiotic investigations of intermodality. This paper therefore shares two semiotic studies of distinct and combined music, English speech and image systems in an animated short film and a promotional filmtrailer. The paper considers the impact of music and voice sound on interpretations of film narrative meanings. A music system relevant to the analysis of filmic emotion is proposed. Examples show how music and intonation contribute meaning to lexical, visual and gestural elements of the cinematic spaces. Also described are relations of divergence and resonance between emotion types in various couplings of music, intonation, words and images across story phases. The research is relevant to educational knowledge about sound, and semiotic studies of multimodality

Teaching About Sound for Multimodal Literacy

The emergence of digital technologies has changed the design of texts and our literate practices so that we now interpret and construct texts which have written, visual, audio and spatial dimensions for making meaning, that is, multimodal texts. Prominent researchers (Kress 2000; The New London Group 2000) argued that literacy should be redefined as multiliteracies to acknowledge the multiplicity of communication channels and media, and the cultural and linguistic diversity that impact on textual meanings. Current definitions of multimodal literacy (Jewitt & Kress 2003) which acknowledge all modes (e.g. image, text, sound, movement) as significant for communication, accordingly require explicit teaching that provides all students with resources for the interpretation and construction of multimodal texts in classrooms. This study contributes to knowledge about the nature and function of the meaning-making resources of sound, which is required to realise multimodal literacy

What are the theoretical relationships between Music, Film Music, and Emotions, and what are the implications for a semiotic analysis of Affect in Narrative Film?

Music is often described as the language of emotions, and film music, it is argued, is the strongest source of emotion in cinematic film. Despite the ubiquity of emotional responses to music-alone and film music, the special relationship between music and emotion almost defies explanation. Questions tend to be asked about whether emotions might reside within the music itself, and about how the listener perceives, experiences and responds to emotions expressed by the music. Theoretical relationships between film music and emotion are informed by both psychological perspectives on emotions per se, and cognitive-psychological research into the music-emotion relationship. As well, film theorists have modelled how film music impacts on the listening/viewing audience (hereafter called the subject) and causes emotional experience. An examination of the interdisciplinary theorising of music and emotion should be able to inform future research that seeks to analyse the relationship between film music and emotion, in narrative film

Semiosis in the Film Soundtrack: Aural Perspective and Social Distance in 'The Queen' Film Trailer

The emergence of digital technologies has changed the design of texts and literate practices so that now texts which have written, visual, audio and spatial dimensions for making meaning are interpreted and constructed. Contemporary digital texts such as television advertisements, film trailers, video and television programs increasingly privilege sound features (speech, music, sound effects) to 'design' meanings and to 'position' listeners towards the interests of the composers of multimodal texts. Sound now takes a significant place alongside language and visual images in the digital texts of our multimodal landscape. This article discusses the educational imperative for schools to begin developing students' knowledge about sound as an integral communicative mode in contemporary digital texts. The article also argues the need for research that builds a rigorous basis for theorising the modal resources of sound, and informs pedagogical practice. One such research perspective in relation to sound is examined in this paper. van Leeuwen's theoretical modelling of sound as a social semiotic, as presented in his book Speech, Music, Sound is outlined, and it informs the exploratory research design reported in the final section of this paper. van Leeuwen's assertion that the conceptual and technical description of semiotic resources of sound facilitates the interpretation of meaning is investigated. Using 'The Queen film trailer' as a model, van Leeuwen's methodology is applied to analyse the semiotic resource of loudness, and interpret the meaning of loudness. The educational implications of this investigation are discussed

Emotions in Filmtrailers: A Social Semiotic Analysis of Wordings, Intonation and Music

The study investigates how the separate and combined resources of wordings, intonation and music are used to construct emotions and shape the interpersonal worlds of characters in six filmtrailers promoting psychological narrative feature films. The study responds to the concerns of educators that much more needs to be known about the interpersonal potential of sound resources in different texts and contexts to meet the demands of contemporary multiliteracies curricula in Australia. The social semiotic study of multimodality is informed by a systemic functional theory of semiosis, which provides for the analysis of meanings made by distinct semiotics in social contexts. A range of analytical methods is used to describe interpersonal semantics within and across narrative phases. An appraisal framework describing evaluative attitudes in English is used as a means of coding the expression of emotions, judgements and appreciations consistently across the wordings, intonation and music of filmtrailer texts. Different types of prosodic realisations are analysed to describe interpersonal motifs created by wordings, intonation and music. The intermodal and intramodal coupling of attitudinal realisations is analysed to describe how wordings, intonation and music contribute in concert to emotional meaning in orchestrations of genre features. The study identified extensive resources used to highlight the emotions, contrasting attitudes and transformed attitudes typical of the genre, to interest global audiences in the feature films. It was found that music and intonation are vital contributors to the expressions of displeasure, disquiet and misery shaping the kinds of emotional situations that are relevant and familiar to audiences. The study extends current approaches to the analysis of multimodality by drawing on complementary theories and descriptions of sound and emotion. It describes the intonation and music resources used to instantiate attitudinal sub/category meanings and their intensification. It explores prosodic realisations relevant to intonation, and exemplifies multimodal construals of emotional tone